Integral channel feature

Integral Channel Features (ICF), also known as ChnFtrs, is a method for object detection in computer vision. It uses integral images to extract features such as local sums, histograms and Haar-like features from multiple registered image channels. This method was highly exploited by Dollár et al. in their work for pedestrian detection, that was first described at the BMVC in 2009.

Overview of the method

1. Compute multiple registered image channels from an input image, using linear and non-linear transformations
2. Extract features such as sums over rectangular channel regions from each channel. The features extracted from various channels are called integral channel features.
3. Train the AdaBoost classifier. Dollár et al. used boosting technique which offers faster learning but training could be done with any of the other available methods such as support vector machine.
4. Finally, trained classifier is used to detect objects

Images and channels

Typically, a "channel" refers to a certain component that defines pixel values in a digital image. A color image, for example is an aggregate of three channels (red, green and blue). The color data of an image is stored in three arrays of values, known as channels. While this definition of a "channel" is widely accepted across various domains, there exists a broader definition in computer vision, which allows one to exploit other features of an image besides the color information. One such definition refers to a channel as a registered map of the original image where the output pixels are mapped to input pixels by some linear or non-transformation. According to this notion of a channel, color channels of an image can be redefined as output images that are obtained by extracting one specific color information point from the input image at a time. Similarly, a channel for a grayscale input image is simply equal to a grayscale input image. The simple MATLAB implementation below shows how color channels and grayscale channel can be extracted from an input image.

I = imread('I_RGB.png'); % input color image

% Output_image = color_channel(I),
% where color channel could be red, green or blue. The three output images
% are extracted from input image as follows
red_channel = I(:, :, 1);
green_channel = I(:, :, 2);
blue_channel = I(:, :, 3);

% Output image = grayscale_image(I).
% Note if input image I was already a grayscale image, grayscale channel
% would have simply been equal to input image, i.e., gray channel = I
gray_channel = rgb2gray(I);

It is clear from the above examples that a channel can be generated by either simply extracting specific information from the original image or by manipulating the input image in some form to obtain the desired channel. Dollár et al. defined a channel generation function as Ω, which can be used to relate a channel (that is, an output image) to the original image as follows.

𝐶 = Ω(𝐼) , where C is the channel and I is an input image

The next section discusses other relatively complex channel types as mentioned in the original paper by Dollár et al. MATLAB implementation is given for some of the channels.

Various channel types

• Color and grayscale: As discussed above, we can easily extract color and grayscale channels from an image. Note that color channels could be RGB, LUV or HSV. LUV color channels have shown to be most informative among all color spaces.
• Linear filters: This is a simple method for generating channels. There are variety of linear filters that allow us to capture different aspects of an image. A few examples are Gabor filter and difference of Gaussians (DoG). Gabor filter and DoG capture edge information and textured-ness of an image. Below is a sample code for implementing DoG in MATLAB.

% Output image = DoG(I)
% Difference of Gaussian applied on input image
H1 = fspecial('gaussian', 25, 0.5); % create a Gaussian with signal 0.5
H2 = fspecial('gaussian', 25, 3); % create a Gaussian with signal 3
DoG_filter = H1 - H2; % create a DoG
image = double(rgb2gray(imread('RGB_1.jpg')));
DoG_channel = conv2(image,DoG_filter,'same'); % convolve DoG with input image

• Nonlinear channels: There are many non-linear channels. The most popular ones are canny edge detector and gradient magnitude. Canny gives the edge information whereas gradient magnitude gives edge strength.
• Pointwise transformations: As the name suggests, pointwise transformations operate on individual pixels. The examples include logarithmic operator and exponential operator. The logarithmic operator enhances low intensity pixels whereas exponential does the complete opposite.
• Gradient histogram: It is a histogram of an image where bins are determined by the gradient angle. Each pixel votes and the weight is determined by its gradient magnitude. Histogram of oriented gradients (HOG) descriptor is a popular descriptor that was developed by Dalal and Triggs.

Note that these channels can be used alone or in combination with each other.

Feature extraction

Once channels are obtained from an input image, various features can be extracted from these channels. These features are called channel features and can be categorized into two main types:

• First-order channel features: Features extracted from a single channel by summing pixels in fixed rectangular regions. These are denoted as ƒ(C). Note that it is computationally less expensive to extract local sums from an integral image rather than from image pixels. In fact, Dollár, et al., used integral images in their work to extract features.
• Higher-order channel features: Features obtained by combining two or more first order channel features. For example, Haar features.

Integral channel feature implementation

The ChnFtrs method allows one to pool features that capture the richness from diverse channels. Dollár, et al. based their experimental results on first order features since there was not much added value by the second order features. The channels are re-computed at multiple scales to extract a pool of channel features that can represent the entire scale space. There is a MATLAB toolbox that can be used as a guidance to implement ChnFtrs method. Further, OpenCV has a complete implementation of ChnFtrs.

Performance

To study the performance of ChnFtrs, Dollár et al. first evaluated the effectiveness of various channels when used individually. The channels studied were histogram of oriented gradients (HOG), gradient histogram channel (Hist), gradient magnitude (Grad), color channels (RGB, HSV, LUV) and grayscale channel. The performance was evaluated in terms of pedestrian detection rates at the reference point of 10 - 4 fppw (false positive per window). HOG turned out to be the most informative channel compared with rest of the channels. The detection rate of HOG was 89%. Further, among the color channels (RGB, HSV and LUV), LUV had the best detection rate of 55.8%. Grayscale channel was least informative with the detection rate of only 30.7%. Next, they evaluated the performance of various channel combinations, which is their proposed method. The combination of LUV, Hist and Grad channels had the highest detection rate of 91.9%. This channel combination was further used in their experiments on INRIA and Caltech datasets.

About 30,000 first order features were used to train AdaBoost classifier. The ChnFtrs + AdaBoost detector was tested on full images from INRIA and Caltech datasets. The performance was compared with 12 other detectors including HOG, which is the most popular method. ChnFtrs outperformed all except LatSvm. The detection rate for ChnFtrs was 86% on INRIA dataset and 60% on a more challenging Caltech dataset.

Further development

The ICF method (ChnFtrs) has been widely exploited by researchers in Computer Vision after the work was initially published by Dollar et al.. In fact, it is now used as a baseline detector due to its proven efficiency and reasonable performance. Several authors have obtained even better performance by either extending feature pool in various ways or by carefully choosing the classifier and training it with a larger dataset. Work by Zhang et al also exploited integral channel features in developing Informed Haar detector for pedestrian detection. They used the same combination of channels as Dollár et al. but were able achieve approximately 20% higher performance than the baseline ChnFtrs method. The added performance was due to the fact that they provided better prior knowledge to their detector. It is also important to note that they used informed Haar-like features, which are second order features according to the terminology described in, whereas Dollár et al. demonstrated their results using first order channel features only, as their analysis showed that second order features barely added 0.6% increase to their detection rate. Further, Benenson et al. were able to increase the detection speed of baseline ChnFtrs method by avoiding the need to resize input image.

ChnFtrs is a versatile method that allows one to extract features from multiple channels, thus allowing to capture diverse information from a single input image. The performance of a base detector developed by Dollár et al. has been shown to be enhanced by adding better prior knowledge and training with a larger dataset.